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ABSTRACT 
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for the five campuses to obtain. Forces shaping the development of 
the Report Card are addressed, including the explaining vs. 
evaluating role of institutional researchers, using formative vs. 
summative evaluations, and system vs. campus concerns. The Report 
Card was released to controversy both on and off the campuses. 
Lessons learned in the process are discussed, including many with 
application to the larger subject of accountability reporting. 
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ABSTRACT 



The rising tide of "accountability" has reached most institutions, often in the form of 
mandated performance measures. The University of Massachusetts system launched a 
legislative "Report Card" project in an effort to pre-empt accountability mandates and improve 
institutional performance. In the process the University developed legislative survey and focus 
group tools; empowered a system-wide University Institutional Research Group; and developed 
peer comparisons and other evaluative tools. The Report Card was released to controversy 
both on and off the campuses; the authors discuss lessons learned in the process, including 
many with application to the larger subject of accountability reporting. 
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INTRODUCTION 

The rising tide of interest in "accountability" has reached most institutions in one form or 
another, but with varying force and effect: some institutions feel they have been inundated by 
legislative reporting mandates — some directly tied to resource allocation — v^hile others are 
only nov^ getting their feet v^et. There is growing consensus, however, that no institution 
receiving public funds can hold back ihe waters indefinitely. 

Accountability demands will not and should not go away. The decisions of those who pay 
for and enjoy the benefits of higher education must be informed by the best insights we can 
offer. As a state legislative oversight committee concluded in a 199?. study of faculty workload 
at a major public university: 

The Committee recognizes that measuring outputs and performance is difficult in 
many state agencies. However, the Committee is surprised that an institution of 
public higher education, which is inherently predisposed toward the transfer of 
knowledge, is seemingly unable to communicate its activities to the wider 
community. 

THE GENESIS OF THE UMASS "REPORT CARD" 

In the face of these trends University of Massachusetts President Michael K. Hooker 
sought to "get ahead of the curve" and demonstrate that a public university could be 
accountable even without an external mandate. He conceived of the UMass Report Card as a 
major annual accountability report for the State Legislature and the public at large, to be 
designed and produced by the University itself. 
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Michael Hooker took office in September of 1992 as the first permanent president since 
a legislative reorganization that added two campuses to the existing three-campus University of 
Massachusetts system. The reorganization devolved substantial governing authority from the 
state-wide Board of Regents (now the Higher Education Coordinating Council) to the UMass 
Board of Trustees. 

President Hooker saw the Report Card as having three principal purposes. First, the 
Report Card was to increase the credibility of the University and of the information it provides 
to state policy makers. Second, Hooker wanted to use the Report Card as a tool to promote 
goal-setting and improvement on the part of the University's campuses. Finally, since the 
Report Card was to be released annually, he hoped that it would demonstrate progress and 
improvement for the whole University over time. The Report Card project was initiated in 
August 1993 and the first edition was released in November 1994. 

DEVELOPMENT 

Early in the development of the Report Card, President's Office staff involved in the 
project met with representatives of Coopers & Lybrand's Higher Education Consulting Services 
to seek advice on shaping the project (Coopers & Lybrand provided this service as part of its 
auditing contract with the University). From that meeting. President's Office staff developed a 
brief survey of members of the Legislature, sent out on Coopers and Lybrand letterhead, 
intended to provide a sense of legislators' highest priorities for UMass. A total of 92 responses 
to the survey were received, representing an overall response rate of 43 percent. 

Almost all the areas of University activity referenced in the survey were seen as 
important by legislators. However, differences emerged when respondents were asked to 
identify their top three priorities for the University: 



• Seventy-seven (77) percent of those surveyed said accessibility/affordability should be one 
of the system's top three priorities, more than any other single item in the survey. Sixty-nine 
(69) percent rated the quality of teaching a top priority for the system. The level of 
consensus evident around the importance of access and teaching placed these two priorities 
in a class by themselves; all other items in the survey garnered substantially weaker support. 

• Post-graduate success was rated a top priority by 28 percent of respondents, and 
retaining/ graduating students was so rated by 17 percent (although both are arguably 
components of the quality of teaching, they were offered in the survey separately). 

• One quarter of those surveyed indicated that efficient use of resources should be one of the 
University's top three priorities, and a quarter chose economic development and impact. 

• The level of consensus fell off rapidly after this point. Only 15 percent of respondents felt 
that quality of research & scholarship and public service to the Commonwealth should be 
top priorities. Similarly, only 9 percent selected diversity of students, faculty and staff. 
However, all of these had been and continue to be clear internal priorities of the UMass 
syvStem. 

Concurrent with the legislative survey, the Assistant Vice President for Academic 
Affairs in the President's Office began meeting regularly with the institutional research directors 
and other staff of the five UMass campuses to discuss the kinds of data that might be available 
and to develop uniform definitions. These individuals had already worked together on a 
number of projects, and had formed what is now the UMass Institutional Research Group 
(UIRG). As the group worked from a long list of possible reporting items to select the data that 
would ultimately be used, several guiding principles emerged: 
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• Use of peer comparisons or other comparative data. The President had articulated a 
desire to include peer comparisons wherever possible. To do this, UIRG relied on previously 
established peer groups for each campus consisting of eight institutions of roughly similar 
mission, size, and program mix. These peer groups had been chosen two years earlier, with 
the assistance of the National Center for Higher Education Management Systems 
(NCHEMS), for a set of financial comparisons (some of which were also included in the 
Report Card). While it would probably have made sense to modify the peer groups for the 
expanded purpose of the Report Card, time did not permit such a process. 

• Graphical display of data with limited text, T^e UIRG decided that the body of the 
Report Card should consist of quantitative data presented in charts, graphs, and pictures, 
with limited amounts of narrative. As a general princ: pie, it was felt that the Report Card 
should be attractive and easily accessible to its intended audience. It was also understood 
that state-level policy makers would be able to devote only limited time and attention to the 
Report Card. The University therefore hired a graphic design firm to assist with the final 
production and layout. 

• Responsiveness to legislative interests as revealed in the survey. The Report Card was 
organized into four sections that roughly correlated to issues highlighted by the legislative 
survey: Opportunity (access/ affordability); Teaching; Public Service, Economic 
Development, and Research; and Resources (efficiency). 

• Feasibility, While many kinds of evaluative information could have been (and were) 
imagined, the UIRG was constrained by the limits of available information, or the limits of 
generating comparable information by all the campuses, especially under a tight timetable. 
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Once a first draft had been assembled, the President's Office sponsored several small 
focus group meetings at which reactions of legislators and legislative staff were solicited. The 
meetings largely affirmed the value of the project and the relevance of the specific data we 
included. Some concrete suggestions for additional data were made and incorporated into the 
final report. 

FORCES SHAPING THE DEVELO . MENT OF THE REPORT CARD 

The development and implementation of the Report Card project were shaped by a 
number of forces at work in the University of Massachusetts system and in higher education 
generally. At each critical juncture, choices were presented that required the institutional 
research staff to acknowledge and account for these forces. In many cases demands and 
interests influencing the project were in competition or even conflict, and the IR staff was 
required to balance these competing interests in order to make it possible for the project to 
continue. 

Each set of competing interests introduced its own tension into the process. In fact, in 
many ways the project was defined by these tensions, and by the strategies employed by the 
institutional research staff to deal with them. While many forces interacted as the project 
unfolded, three areas of clear and persistent tension had a powerful impact on the final form 
and content of the Report Card. 

Explaining vs. Evaluating 

The institutional research craft frequently places its practitioners in the position of 
altempting two jobs at once. On the one hand, institutional research staff often stand at the 
"front door" of an institution, describing and explaining the institution's activities to many 
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audiences. Often, these audiences have little if any familiarity with the institution or even 
higher education, and the descriptive role of the institutional research professional becomes 
both challenging and important to an institution's success. 

An example of this descriptive, or explanatory, role can be seen in the growing interest in 
faculty workload. Legislators, trustees and others often demonstrate a keen interest in the 
general subject, but may not be well-versed in the complexities of the issue. This is especially 
true with respect to research universities, where the multiplicity of and variation among faculty 
roles is often poorly understood even on the campus itself. Before one can intelligently assess 
faculty workload it is necessary to understand it. With faculty roles ranging from clinical 
medical faculty who may not generate a single student credit hour to part-time instructors hired 
on a piece-work basis to staff lower-division or general education courses, the appropriate 
workload consistent with institutional policies and expectations is not always obvious to an 
outside observer. In such a case, it is often the institutional research professional who must set 
the context by making reference to relevant policies, arraying information in ways that respect 
institutional expectations, and placing institutional practice within the framework of other, 
similar institutions. 

On the other hand, institutional research staff are important — and often the sole — 
evaluators of institutional practice. To employ the same example, it is often the institutional 
research office that collects, reports and interprets faculty workload data. Depending on 
institutional culture and tradition, the institutional research office may make intra- or inter- 
campus comparisons, place institutional data in the context of national norms (such as the 
distribution of effort reported from the National Survey of Post-Secondary Faculty [NSOPF]), 
or develop prescriptive policy papers and analyses. 



These roles are not necessarily in conflict. One can explain without evaluating, and vice 
versa, with the choice or emphasis of roles often depending upon the audience being addressed. 
With external audiences the explanatory role was once the most common, while the evaluative 
role was most meaningful inside the institution. This separation of roles tended to keep them 
out of conflict. 

Increasingly, however, as higher education has come under greater and greater external 
scrutiny, the need to perform both roles simultaneously has grown, as well. Where once it was 
accepted that evaluation was principally the province of institutional or system administrators, 
we now see much greater interest in exploring and even shaping evaluation on the part of 
trustees, legislators and others. Evaluative data are often placed directly into the hands of 
external audiences — sometimes by legislative mandate — and the ability of institutions to 
regulate the uses to which such data ai ^ put has substantially been diminished. Meanwhile, the 
explanatory role has not disappeared; if anything, it has become more important as evaluative 
data travels further afield. The tension is clearly evident in the Report Card. What be^an as a 
series of relatively straightforward, largely graphic representations of data rapidly accumulated 
notes, caveats and, finally, "President's Comments" that sought to explain the meaning of the 
measures. 

But performing the explanatory and evaluative roles simultaneously can be quite 
difficult. The campus community frequently wants the institutional research professional to 
"explain away" analyses that raise questions or inconsistencies. From inside the institution, the 
institutional research professional's explanatory role can be seen as critical in deflecting 
unwelcome interest or criticism from the outside. But external constituencies tend to have little 
patience for the explanatory role, seeing it for what it sometimes is: the role of the institutional 
apologist who is at best unresponsive and at worst deceptive. From the outside, the value of 
the institutional research professional lies in his or her role as a dispassionate evaluator, a 



speaker of truths that others will not utter. When audiences of both kinds — internal and 
external — are involved, the conflict between expectations can become impossible. 

The UMass Report Card project placed the institutional research staff from the 
campuses into the midst of this conflict. On the one hand, they were expected to manage the 
process of creating an accountability tool with a decidedly "outside" orientation, and to stress 
the evaluative role. The University President made it clear that he wanted the Report Card to 
have high credibility among its intended audience of legislators, officials in the executive branch, 
and others. On the other hand, aside from the Assistant Vice President who coordinated the 
project, all those working on the project were staff members from the five UMass campuses. 
They felt varying degrees of pressure to justify, qualify, and buffer the impact of the evaluative 
data. To their home campuses, the explanatory role was most important. 

Formative vs. Summative 

The "evaluative" role of the institutional research professional involves its own tension, 
between the formative and summative dimensions ui' Ll\e enterprise. The former focuses on 
improvements in practice, the kind of "institution-building" that any adaptive organization 
must engage in. The focus is on understanding how the institution accomplishes its work, not 
on judging its value, institutional research professionals also engage in summative evaluation, in 
which the goal is to assess the performance of the institution so as to draw conclusions of one 
kind or another. The summative side of institutional research work is frequently associated 
with resource allocation processes, when a decision must be made regarding continued 
investment in progratns. 

As external scrutiny of colleges and universities has increased, so too has the emphasis 
placed on summative evaluation. With rare exception, legislators and others external to the 



campus c re interested in making judgments about institutions, not in gaining insights into 
institutional practice. In part this summative emphasis is rooted in a healthy respect for 
differing roles: excessive interest in formative evaluation on the part of legislators or state-v^ide 
bocirds, for example, could be a sign of inappropriate "micro-management" on the part of 
policy makers. More important, however, is the simple fact that the growing competition for 
public funds uemands more — and more precise — funding decisions. The increasing difficulty 
of these decisions creates a market for information to support them. Almost by definition, 
information that supports decision-making tends to emerge from summative, rather than 
formative, evaluation. 

The tension between formative and summative evaluation was manifest in the UMass 
Report Card project. A certain amount of anxiet}^ naturally accompanies the prospect of one/s 
organization being judged from the outside. The anxiety of institutional research professionals, 
however, tends to be heightened by virtue of their understanding of the weakness of the data 
available for summative judgments. The kind of information available to most institutions is 
simply not up to the task of supporting fine judgments, and yet a project like the Report Card 
seems to invite such judgments. The campus institutional research professionals found 
themselves struggling with the strong desire of the University President to "grade" the 
institution. 

System vs. Campus 

Institutional Research professionals can also be subject to a related but distinct set of 
tensions involving the relationships between a University system administration and the 
campuses within the system. Even in the most congenial and well-managed university system 
differences exist between the aspirations and interests of the campuses and the central 
administration. These differences can be rooted in many factors: interinstitutional competition. 
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resource allocation and control, and differing perceptions of institutional traditions and 
directions, to name but a few. 



Differences of this kind are not inherently damaging, and many would argue that they 
are to be anticipated and respected in a vibrant university system. Responsibility for dealing 
with the J.'esulting tension, however, usually resides with the campus CEOs and the system 
administrator and central staff. Rarely is the "middle management" of an institution asked to 
engage in the kind of diplomacy that successful campus-system relationships rely on. 

The UMass Report Card project, however, made diplomats-without-portfolio of each of 
the participating institutional research staff members. Each was asked, in effect, to represent 
the interests of both his or her campus and the University system. Since the fundamental 
premises of the Report Card project originated in the system office, with relatively little 
interaction with the campuses, this form of diplomacy proved to be quite difficult. 

As a relatively new system, the University of Massachusetts was still working through a 
number of campus-sysfem and campus-campus issues when the Report Card project was 
initiated. One set of issues related to the diverse nature and traditions of the UMass campuses. 

Another set of issues centered on the production and analysis of institutional data. 
Accompanying the reorganization of the University were new demands for information from the 
President's Office and trustees. However, there was no institutional research function and few, 
if any, centralized reporting capabilities. The President's Office assumed a coordinating role 
and representatives from each of the campuses, some institutional research professionals and 
others new to the field, worked together to develop reporting systems for the University. When 
the Report Card project began, the institutional research function had been assumed by the 
assistant vice president for academic affairs, who served as coordinator of the institutional 
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research group. The capacities within the system differed greatly from campus to campus. 
Two of the campuses had established institutional research offices with at least three 
professional and support staff members and fairly sophisticated reporting capabilities. At the 
other extreme, one campus had no institutional research staff nor established reporting systems 
and had no choice but to estimate if the data were not available. To this day, the campuses a^-e 
struggling to develop common data definitions. 

The difficulty of merging campus and system perspectives is illustrated in the section of 
the Report Card dealing with undergraduate education. Two graphs are displayed; one was a 
distribution of course sections shown by type of instructor, and the other was a distribution of 
course sections by section size. In operationalizing the definition of a "course section," the 
UIRG agreed that independent study and other types of individualized instructional 
arrangements would not be included. It soon became evident, however, that only one campus 
could identify such sections through its course-numbering system. In the end, this type of 
instruction was approximated by defining a "course section" as a section enrolling three or more 
students. 

The Report Card was designed to report on the University system and to demonstrate 
the impact of the University on the state. This limited the campuses' ability to showcase their 
programs or highlight successes in specific areas. The five campuses, aside from being 
universities, differ in size, mission and program mix. However, from the system perspective it 
was sometimes preferable to report data in the aggregate. For example, one chart focused on 
the diversity of students attending UMass. What was not said was that the urban campus had 
a disproportionately large number of students of color, and strongly affected the profile for the 
system. Similarly, another chart on doctoral degree production compared the number of degrees 
awarded in the system to those awarded by the state's private institutions. What was not said 
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was that the majority of doctorates were awarded by the research campus in the system. 
Clearly, the data would have been reported differently if the perspective were the campuses. 



Data presentation varied depending on the topic and availability of data. Some data 
were aggregated and presented at the system level; some were presented individually for the 
four campuses, some were presented for only one campus and finally, some compared the 
UMass campuses to a group of peer institutions. The peer groups had been selected three years 
earlier for a set of financial comparisons, specifically sources of revenue and spending per 
student. The campuses, to varying degrees, were uncomfortable with using the same peer group 
for non-financial comparisons; however, the schedule for the Report Card did not allow time for 
identifying a different set of peers. 

RELEASE AND AFTERMATH 

The release of the Report Card received widespread attention in the state's print media. 
The report was characterized as "candid" and "eye-opening" by the Boston press. Legislators 
gave the University high marks for producing the report, although it is not obvious that many of 
them took the time to absorb its contents despite the efforts to make it easy to digest. 

However, reaction to the Report Card proper is difficult to assess because most of the 
press attention focused on a remark the President made in response to a Boston Globe reporter's 
question on the day it was released. The reporter asked, "if this is a report card, what overall 
grade would you give the University?" After a moment's hesitation, the President replied, "C- 
plus." The front-page Globe headline the next day read: "UMass receives average grade from vts 
chief." 
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Many UMass faculty, staff, and students were outraged by this remark, and months 
later, despite an explanation and apology from the president circulated to all faculty, the 
remark continues to rankle. There is also evidence of some disturbing unintended consequences, 
such as reluctance on the part of prospective students, parents, donors, or business associates 
of the University to invest their money and confidence in a "C+" institution. At the same time, 
faculty and staff who have actually read the Report Card itself are considerably less offended 
by its contents than by the grade. 

One gratifying outcome of the release of the Report Card is that it served as the basis of 
a lively, candid, and stimulating discussion among the Board of Trustees, the President, and the 
five campus Chancellors. The discussion focused on the policy implications of the information 
presented and the tradeoffs and judgments involved in seeking improvement in specific areas. 
Staff who were present at this discussion felt it was one of the best Trustee meetings they had 
ever attended. 

One thing the publication of the Report Card did not do is pre-empt the demand for 
greater and greater amounts of similar "accountability" data from state policy makers. Just two 
months after the Report Card was released, the state's Secretary of Administration and Finance 
sent a letter to the chair of the UMass Board of Trustees asking the University to submit an 
annual report with "performance benchmarks" on approximately 35 indicators, many of which 
were used in the Report Card. Several months after that, the House passed a budget including 
a long list of "Program Performance Measures" to be submitted each semester. Some were 
clearly taken directly from the Report Card; others from the Secretary's letter; and others were 
contributed by the Ways & Means Committee's higher education budget analyst. And the 
Higher Education Coordinating Council appointed a Task Force on Higher Education Goals and 
Objectives, Performance Measures and Accountability which is developing a system-wide 
template for accountability reporting. 




Planning has already begun for the next version of the Report Card, which is to be 
released in March 1996 (on the theory that a spring release would allow inclusion of more recent 
data and have greater impact during the annual state budget debate). In the meantime, the 
President asked each campus to stipulate one, three, and five-year targets for improvement in 
five areas highlighted in the Report Card: SAT scores of entering freshmen, retention rates, 
graduation rates, volume of sponsored research, and dollars raised through fund raising. These 
targets are being monitored by the Board of Trustees. 

LESSONS LEARNED 

Having completed the first round of the Report card project, the authors have 
formulated some tentative observations, many of which have application to the broader subject 
of accountability reporting. While the developm * of the Report Card involved staff from the 
system office and all five UMass campuses, it is important to note that these observations 
reflect the judgments of the authors only. 

1 . It is important to recognize the tensions that naturally occur in the context of accountability 
reporting, and to set realistic expectations for the role institutional research professionals 
play. Regardless of the structure of the project, campus-based institutional research officers 
cannot fully step out of their campus roles and institutional research professionals in general 
will see the need to balance evaluation with explanation and to resist over-emphasis on 
summative evaluation. 

2. In a multi-campus system setting, the success of a project of this kind depends critically on 
the existence of strong working relationships among campus and system institutional 
research professionals. Each member of the team needs to understand the traditions and 
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politics of the other campuses and of the system itself. Given the enormous tensions at 
work m a project of this kind, it is not realistic to expect to build good working relationships 
where there is no existing foundation. UMass was fortunate in that such relationships had 
been forged from the first days of the five-campus system. 

3. It is important to recognize that accountability projects are essentially political, not 
technical, exercises. Despite the best laid plans of institutional research professionals (and 
institutional leaders) it is impossible to predict how the evidence will be read or what will 
capture the headlines. The handling of the release of a document like the Report Card is as 
or more important than the report itself. 

4. Be responsive to legislative concerns. The survey of members of the Legislature and the 
focus groups seeking legislative reaction were important in building support for the Report 
Card, even if the final product was not closely examined. 

5. Use peer comparisons or other comparative data. To provide a context for evaluating 
performance, the UMass Report Card included peer comparisons wherever possible. The 
investment of time and effort necessary to develop reliable peer data is substantial, but its 
presence adds a dimension to accountability reporting that cannot be achieved in any other 
way. 

6. Be honest. The indicators chosen for the Report Card revealed both good and bad news. 
This imparted a sense of balance that contributed to the credibility of the project. 

7. Accept the reality that higher education continues to lack meaningful quantifiable measures 
for many important aspects of its mission, notably teaching and learning and public service. 
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This often requires the use of "proxies" of dubious value (such as the use of GRE scores of 
graduating seniors as a measure of "the quality of instruction"). 



8. Building and maintaining systems for institutional accountability like the Report Card are 
immensely labor- and time-intensive activities. Very little "routine" v^ork can be 
accomplished at the same time, especially in the developmental stages. 

9. Think about how an accountability project will be perceived from the outside. If it is billed 
as a "report card," the expectations that a "grade" will result will be almost overwhelming. 
If it is intended to be more formative or explanatory, that message should be conveyed in 
the style and form of the document. 

10. Recognize that, at least in today's environment, one cannot "pre-empt" external interest in 
accountability. The interest is latent, and institutional efforts can actually give this interest 
form and substance. There is some evidence that the UMass Report Card actually 
stimulated interest in accountability data. Nonetheless, by "going first" it may be possible 
for an institution to set productive terms for the debate and avoid the problems that arise 
when one is reacting to unforeseen outside interest. 

11. Remember that accountability projects may serve other purposes. The UMass Report Card, 
for example, resulted in a number of benchmarks that will be useful for institutional 
purposes. The process of developing consistent data definitions within the new University 
of Massachusetts system was clearly accelerated by the demands of the Report Card 
project. And, by entering the fray, one learns what works and what does not. This can be 
valuable knowledge given the likelihood that higher education is entering a period of 
prolonged and intense scrutiny. 



